Neural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuning

Model-free deep reinforcement learning algorithms have been shown to be capable of learning a wide range of robotic skills, but typically require a very large number of samples to achieve good performance. Model-based algorithms, in principle, can provide for much more efficient learning, but have proven difficult to extend to expressive, high-capacity models such as deep neural networks. In this work, we demonstrate that medium-sized neural network models can in fact be combined with model predictive control (MPC) to achieve excellent sample complexity in a model-based reinforcement learning algorithm, producing stable and plausible gaits to accomplish various complex locomotion tasks. We also propose using deep neural network dynamics models to initialize a model-free learner, in order to combine the sample efficiency of model-based approaches with the high task-specific performance of model-free methods. We empirically demonstrate on MuJoCo locomotion tasks that our pure model-based approach trained on just random action data can follow arbitrary trajectories with excellent sample efficiency, and that our hybrid algorithm can accelerate model-free learning on high-speed benchmark tasks, achieving sample efficiency gains of 3-5x on swimmer, cheetah, hopper, and ant agents. Videos can be found at https://sites.google.com/view/mbm

Body Lift and Drag for a Legged Millirobot in Compliant Beam Environment

Much current study of legged locomotion has rightly focused on foot traction forces, including on granular media. Future legged millirobots will need to go through terrain, such as brush or other vegetation, where the body contact forces significantly affect locomotion. In this work, a (previously developed) low-cost 6-axis force/torque sensing shell is used to measure the interaction forces between a hexapedal millirobot and a set of compliant beams, which act as a surrogate for a densely cluttered environment. Experiments with a VelociRoACH robotic platform are used to measure lift and drag forces on the tactile shell, where negative lift forces can increase traction, even while drag forces increase. The drag energy and specific resistance required to pass through dense terrains can be measured. Furthermore, some contact between the robot and the compliant beams can lower specific resistance of locomotion. For small, light-weight legged robots in the beam environment, the body motion depends on both leg-ground and body-beam forces. A shell-shape which reduces drag but increases negative lift, such as the half-ellipsoid used, is suggested to be advantageous for robot locomotion in this type of environment.Comment: First three authors contributed equally. Accepted to ICRA 201

Rapid inversion: running animals and robots swing like a pendulum under ledges.

Escaping from predators often demands that animals rapidly negotiate complex environments. The smallest animals attain relatively fast speeds with high frequency leg cycling, wing flapping or body undulations, but absolute speeds are slow compared to larger animals. Instead, small animals benefit from the advantages of enhanced maneuverability in part due to scaling. Here, we report a novel behavior in small, legged runners that may facilitate their escape by disappearance from predators. We video recorded cockroaches and geckos rapidly running up an incline toward a ledge, digitized their motion and created a simple model to generalize the behavior. Both species ran rapidly at 12-15 body lengths-per-second toward the ledge without braking, dove off the ledge, attached their feet by claws like a grappling hook, and used a pendulum-like motion that can exceed one meter-per-second to swing around to an inverted position under the ledge, out of sight. We discovered geckos in Southeast Asia can execute this escape behavior in the field. Quantification of these acrobatic behaviors provides biological inspiration toward the design of small, highly mobile search-and-rescue robots that can assist us during natural and human-made disasters. We report the first steps toward this new capability in a small, hexapedal robot

Learning Image-Conditioned Dynamics Models for Control of Under-actuated Legged Millirobots

Millirobots are a promising robotic platform for many applications due to their small size and low manufacturing costs. Legged millirobots, in particular, can provide increased mobility in complex environments and improved scaling of obstacles. However, controlling these small, highly dynamic, and underactuated legged systems is difficult. Hand-engineered controllers can sometimes control these legged millirobots, but they have difficulties with dynamic maneuvers and complex terrains. We present an approach for controlling a real-world legged millirobot that is based on learned neural network models. Using less than 17 minutes of data, our method can learn a predictive model of the robot's dynamics that can enable effective gaits to be synthesized on the fly for following user-specified waypoints on a given terrain. Furthermore, by leveraging expressive, high-capacity neural network models, our approach allows for these predictions to be directly conditioned on camera images, endowing the robot with the ability to predict how different terrains might affect its dynamics. This enables sample-efficient and effective learning for locomotion of a dynamic legged millirobot on various terrains, including gravel, turf, carpet, and styrofoam. Experiment videos can be found at https://sites.google.com/view/imageconddy

Detection of Slippery Terrain with a Heterogeneous Team of Legged Robots

Legged robots come in a range of sizes and capabilities. By combining these robots into heterogeneous teams, joint locomotion and perception tasks can be achieved by utilizing the diversified features of each robot. In this work we present a framework for using a heterogeneous team of legged robots to detect slippery terrain. StarlETH, a large and highly capable quadruped uses the VelociRoACH as a novel remote probe to detect regions of slippery terrain. StarlETH localizes the team using internal state estimation. To classify slippage of the VelociRoACH, we develop several Support Vector Machines (SVM) based on data from both StarlETH and VelociRoACH. By combining the team’s information about the motion of VelociRoACH, a classifier was built which could detect slippery spots with 92% (125/135) accuracy using only four features

NYMPH: A multiprocessor for manipulation applications

The robotics group of the Stanford Artificial Intelligence Laboratory is currently developing a new computational system for robotics applications. Stanford's NYMPH system uses multiple NSC 32016 processors and one MC68010 based processor, sharing a common Intel Multibus. The 32K processors provide the raw computational power needed for advanced robotics applications, and the 68K provides a pleasant interface with the rest of the world. Software has been developed to provide useful communications and synchronization primitives, without consuming excessive processor resources or bus bandwidth. NYMPH provides both large amounts of computing power and a good programming environment, making it an effective research tool

Wearable Microfluidic Diaphragm Pressure Sensor for Health and Tactile Touch Monitoring

Flexible pressure sensors have many potential applications in wearable electronics, robotics, health monitoring, and more. In particular, liquid-metal-based sensors are especially promising as they can undergo strains of over 200% without failure. However, current liquid-metal-based strain sensors are incapable of resolving small pressure changes in the few kPa range, making them unsuitable for applications such as heart-rate monitoring, which require a much lower pressure detection resolution. In this paper, a microfluidic tactile diaphragm pressure sensor based on embedded Galinstan microchannels (70 µm width × 70 µm height) capable of resolving sub-50 Pa changes in pressure with sub-100 Pa detection limits and a response time of 90 ms is demonstrated. An embedded equivalent Wheatstone bridge circuit makes the most of tangential and radial strain fields, leading to high sensitivities of a 0.0835 kPa^(−1) change in output voltage. The Wheatstone bridge also provides temperature self-compensation, allowing for operation in the range of 20–50 °C. As examples of potential applications, a polydimethylsiloxane (PDMS) wristband with an embedded microfluidic diaphragm pressure sensor capable of real-time pulse monitoring and a PDMS glove with multiple embedded sensors to provide comprehensive tactile feedback of a human hand when touching or holding objects are demonstrated